Dynamically structuring, updating and interrelating representations of visual and linguistic discourse context

The fundamental claim of this paper is that salience—both visual and linguistic—is an important overarching semantic category structuring visually situated discourse. Based on this we argue that computer systems attempting to model the evolving context of a visually situated discourse should integrate models of visual and linguistic salience within their natural language processing (NLP) framework. The paper highlights the importance of dynamically updating and interrelating visual and linguistic discourse context representations. To support our approach, we have developed a real-time, natural language virtual reality (NLVR) system (called LIVE, for Linguistic Interaction with Virtual Environments) that implements an NLP framework based on both visual and linguistic salience. Within this framework saliency information underpins two of the core subtasks of NLP: reference resolution and the generation of referring expressions. We describe the theoretical basis and architecture of the LIVE NLP framework and present extensive evaluation results comparing the system’s performance with that of human participants in a number of experiments.  2005 Elsevier B.V. All rights reserved. ✩ The authors would like to thank Hans Kamp, Laurent Romary and our three anonymous reviewers for valuable comments and feedback. * Corresponding author. E-mail address: [email protected] (J. Kelleher). 1 This work was partly supported by the EU FP6 IST Cognitive Systems Integrated project Cognitive Systems for Cognitive Assistants (CoSy) FP6-004250IP. 0004-3702/$ – see front matter  2005 Elsevier B.V. All rights reserved. doi:10.1016/j.artint.2005.04.008 J. Kelleher et al. / Artificial Intelligence 167 (2005) 62–102 63